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U nder the No Child Left Behind 
(NCLB) Act, schools or local 
education agencies that haven’t 
made adequate yearly progress (AYP) 
for three or more consecutive years 
must provide supplemental educa- 
tional services to eligible students. 

In the Northwest region, each state 
has successfully complied with this 
requirement. However, the states now 
face the challenge of monitoring and 
evaluating their supplemental service 
providers. 

The law allows flexibility in design- 
ing and implementing this process, 
but developing a set of successful 
practices is just the beginning. Ensur- 
ing that eligible students measurably 
benefit from supplemental educa- 
tional services will require states to 
be innovative in fulfilling their 
responsibility. 

This Topical Summary focuses 
on evaluating supplemental service 
providers, but some general inform- 
ation about the requirements 
is needed to provide context. The 
following information is summarized 
from NCLB — available at 
www.ed.gov/nclb/landing.jhtml — 
and the U.S. Department of Educa- 
tion’s Supplemental Educational 
Services Non-regulatory Guidance, 
www.ed.gov/policy/elsec/guid/ 
suppsvcsguid.doc. 



Defining 

Supplemental 

Educational 

Services 

Supplemental educational services 
are academic interventions such as 
tutoring and remediation that take 
place outside the regular school day. 
Supplemental educational services 
must be consistent with the content 
and instruction used by the local edu- 
cation agency (LEA) and must align 
with the state education agency’s 
(SEA) academic standards. In addi- 
tion, supplemental services must be 
of high quality, research-based, and 
designed to increase academic 
achievement. 

Students eligible for supplemental 
services attend Title I schools that 
haven’t made AYP for three consecutive 
years and opt to stay at the school 
rather than enroll in another one under 
the choice provisions of NCLB. Supple- 
mental service providers apply to the 
SEA, which creates a statewide list of 
approved public and private providers. 
Local education agencies must notify 
parents that their child is eligible to 
receive services and give them a list of 
approved local providers. 
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Responsibilities 
of the States 

States are required to do the following: 

• Consult with parents, teachers, 
LEAs, and interested members of the 
public to identify a large number of 
supplemental service providers so 
parents have a wide variety of choices 

• Broadly disseminate an annual 
notice to potential providers that out- 
lines the process for obtaining approval 

• Develop and apply objective crite- 
ria for approving potential providers 
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• Maintain an updated list of 
approved providers 

• Give school districts a list of avail- 
able approved providers in their gen- 
eral geographic location 

• Develop, implement, and publicly 
report on standards and techniques for 
monitoring the quality and effectiveness 
of services offered by approved 
providers 



• Provides services that are consis- 
tent with the instructional program of 
the LEA and with state academic con- 
tent and achievement standards 

• Is financially sound 

• Provides supplemental educa- 
tional services consistent with applica- 
ble federal, state, and local health, 
safety, and civil rights laws 

At least once a year, the state must 
update its list and notify current and 
potential providers of the procedures 
to be used for approval. 

Local education agencies have their 
own responsibilities under NCLB, but 
they must also assist the SEA with gath- 



Surveying the parents and teachers 
of children receiving supplemental 



educational services will increase the 



reliability and utility of the evaluation. 



The relationship between the 
process for approving providers and 
evaluating them as they deliver services 
will be discussed later in this Topical 
Summary. States are encouraged to be 
innovative and flexible in the approval 
process, with strong participation from 
LEAs, parents, teachers, and other 
stakeholders. However, NCLB requires 
SEAs to ensure that each approved 
provider meets the following criteria: 

• Has a demonstrated record of 
effectiveness in improving student aca- 
demic achievement 

• Uses instructional strategies that 
are high quality, research-based, and 
designed to increase student academic 
achievement 



ering information to evaluate and mon- 
itor supplemental service providers. 
Specifically, LEAs must: 

• Notify parents, at least annually, 
about the availability of services 

• Help parents choose a provider, 
if requested 

• Determine which students can 
receive services, if all students can’t 
be served 

• Enter into an agreement with 
a provider selected by parents of an 
eligible student 

• Assist the SEA in identifying 
potential providers within the LEA 

• Protect the privacy of students 
who receive supplemental educa- 
tional services, according to the 



Family Educational Rights and 
Privacy Act. 

It is important to note that since 
NCLB was authorized in 2001 , only a 
relatively small number of schools have 
been identified as not making AYP. Con- 
sequently, the need for providers is not 
as great as it will be in later years. 

In its report, From the Capital to 
the Classroom: State and Federal 
Efforts to Implement the No Child 
Left Behind Act, the Center on Educa- 
tion Policy states “since supplemental 
education services are only one part of 
the educational process that could 
affect learning and achievement, it will 
be difficult to identify whether a partic- 
ular service actually improves student 
achievement, and if so, to what extent. 
States face a challenge in figuring out 
how to isolate this part of the educa- 
tional process.” 

The Council of Chief State School 
Officers has drafted an SEA toolkit on 
supplemental educational services, 
which is available at www.ccsso.org/ 
content/pdfs/SSPToolkit.pdf. This 
toolkit offers many resources for 
approving supplemental service 
providers, but the draft document 
doesn’t directly address the need to 
evaluate providers once they are 
approved. The process for approving 
applications can certainly inform eval- 
uation, but it is more concerned with 
technical compliance. Evaluating an 
operating service provider requires 
data from multiple sources. Also, dur- 
ing the approval process, the onus is 
on the provider to meet application 
requirements; during evaluation, the 
onus is on both the provider and the 
LEA to report to the SEA both compli- 
ance by providers and indicators of 
quality and effectiveness. 

In formulating provider evaluation 
plans, SEA personnel should consult 
the CCSSO toolkit. The Resources 
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section (Page 9) may also be helpful 
as you begin the process of evaluating 
supplemental service providers. 

Evaluation Issues 

To develop successful practices and 
a research base for evaluating supple- 
mental educational service providers, 
SEAs and LEAs should start with a look 
at program evaluation research, exist- 
ing Title I monitoring practices, and 
after-school/out-of-school time pro- 
grams. Using information from existing 
knowledge bases is desirable for sev- 
eral reasons: providers will be 
recruited from existing after-school and 
out-of-school time programs and will 
likely be familiar with that evaluation 
process; using common vocabulary and 
language will help facilitate reporting; 
and there is an existing record of suc- 
cessful and unsuccessful programs. 

Deciding what to use from these 
sources depends on the type of 
provider, the services offered, and the 
specific objectives and outcomes con- 
tained in the contract with the provider 
and the parents of eligible students. 
Also, there are several key components 
that must be considered when evaluat- 
ing supplemental service providers. 
These include complying with NCLB 
and other laws; documenting student 
achievement outcomes; determining 
stakeholders’ satisfaction with the 
providers; assessing the extent of com- 
munication and collaboration among 
key parties; ensuring appropriate 
administration by the LEA; and making 
certain that student needs are met. 

It is important to point out the 
difference between quality and/or 
effectiveness of providers and technical 
comphance. The SEA may approve 
providers based on their comphance 
with NCLB. However, LEAs — working 



with stakeholders — must develop 
quality and effectiveness indicators for 
evaluating the providers. As LEAs are 
required to develop contracts with 
providers, the contract performance 
objectives can be used to write the 
quality/effectiveness indicators. 

Research-Based Practices 
for Student Achievement 

The NCLB Act requires supplemental 
educational services to be “high quality 
and research based.” Scientifically- 
based research is defined as “research 
that involves the application of rigor- 
ous, systematic, and objective proce- 
dures to obtain rehable and vahd 
knowledge relevant to education activi- 
ties and programs” (Beghetto, 2003) . 
During the application process, the 
provider should submit information 
about the relevant interventions that 
it plans to use with the student. The 
Council of Chief State School Officers’ 
SEA Toolkit on Supplemental Educa- 
tional Services provides excellent 
tools for determining if the evidence 
submitted in the provider application 
meets this requirement. These indica- 
tors can also be used to evaluate 
providers once they are operating. 

For example, if a supplemental service 
provider tutors a student to increase 
achievement in reading, the provider 
should utilize the findings from the 
National Reading Panel and other suc- 
cessful practices in reading research. 
Although standardized test scores are 
useful in measuring a student’s 
achievement, there is a danger when 
those are the only criteria used to eval- 
uate an educational program or sup- 
plemental educational service provider 
(Beswick, 1990). 

When an SEA evaluates the 
provider, the provider should be 



responsible for reporting the 
effectiveness of the intervention and 
its research base. States can ensure 
the accuracy of the provider’s claims 
of research-based interventions by 
using resources such as the 
What Works Clearinghouse at 
www.w-w-c.org/index.html or 
the National Reading Panel at 
www. nationalreadingpanel. org/. 

Program Evaluation 

The field of program evaluation offers 
a long and lengthy research base on 
which to draw. The Program Evalua- 
tion Standards (1995), an ERIC digest 
from the ERIC Clearinghouse for Assess- 
ment and Evaluation, states that sound 
program evaluation should have four 
basic attributes: utility, propriety, feasi- 
bility, and accuracy. 

• Utility means the evaluation 
will serve the information needs of 
intended users. State education agen- 
cies should ensure the process for 
evaluating supplemental service 
providers is not cumbersome and will 
gather the necessary information. The 
process should be accomplished in a 
relatively short time period since lists 
of approved providers must be updated 
annually. 

• Propriety ensures that evalua- 
tions are conducted legally, ethically, 
and with due regard to the welfare and 
privacy of both the student and the 
provider. 

• Feasibility means that the evalu- 
ation will be realistic and within the 
means of the SEA. 
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Evaluation Prompts 



Evaluation 

Prompt 


Evidence/ 

Indicators 


Quality 

Indicator 


Compliance 

Indicator 


Is the provider a legal entity and approved to provide services? 


Business license, 501(c) 3 
determination letter, etc. 




Yes 


Does the provider offer services consistent with applicable federal, state, 
and local laws? 






Statement of 
assurances 


Is the provider financially sound? 


Financial statements, au- 
dits performed by state 
auditor, bank letters of 
reference 






Are the requirements of the Family Educational Rights and Privacy Act fol- 
lowed in till transactions and communications while planning, delivering, and 
monitoring supplemental educational services? 






Survey 

instruments 


If the provider is a faith-based organization, are its services secular, neutral, 
and nonideological? How does the faith-based organization maintain financial 
records on the supplemental services contract? 








Are the services consistent with the LEA's instructional program and aligned 
with the state's content and academic achievement standards? 




Sample descriptions of 
services, surveys 




If eligible students have limited English proficiency or disabilities, does the 
provider contract contain provisions to address these needs as appropriate? 




Student pre- and post- 
data, writing samples 




Has the provider developed specific achievement goals for the student in con- 
sultation with tire parent, child, and LEA? 


Provider assessments, 
school assessments, port- 
folios of student work 


Contract language, con- 
tract performance objec- 
tives 




How does the provider measure and document student progress? 




Criterion-referenced as- 
sessments, student port- 
folios, district assessment 
data 




Does the provider regularly inform parents and teachers of the student’s 
progress? 




Letters, Web page com- 
munications, individual 
contact 




Has the provider developed and adhered to a timetable for improving the 
student's achievement in consultation with the parents and LEA? 








Are the methods used to deliver supplemental educational services based on 
research? 









Evaluation Prompts 
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Evaluation 

Prompt 


Evidence/ 

Indicators 


Quality 

Indicator 


Compliance 

Indicator 


LEA . . . 








Has the LEA notified parents at least annually of the availability 
of services? 


Legal notices, newslet- 
ters, Internet sites 






What mechanisms are in place to assist parents in choosing 
a provider? 








How has the LEA determined which students receive services if 
demand is greater than available resources? 








What mechanism is in place for the LEA to help the state identify 
potential providers? 








Does the agreement between the LEA and the provider contain 
the following: 








1. Proper accounting of funds for supplemental educational services 








2. Proper accounting of payments for supplemental services 








3. Specific achievement goals for the student developed in consulta- 
tion with the student’s parents 








4. A description of how the student’s progress will be measured, 
and how parents and teacher will be informed 








5. A timetable for improving the student’s achievement 








6. Provisions for terminating the agreement if the provider fails 
to meet student progress goals and timetables 








7. Provisions for payment of services 








8. Provisions for protecting student privacy and nondisclosure 
of information without parent’s written consent 








9. Assurances that services will meet all applicable health, safety, 
and civil rights laws 
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• Accuracy ensures the evaluation 
reveals adequate information to deter- 
mine merit. 

All these attributes help the SEA 
design an evaluation system that guards 
against hasty judgments, lets providers 
know exactly how they will be evalu- 
ated, and links expected outcomes to 
increased student achievement. 



Family Research Project and the 
Finance Project’s Documenting 
Progress and Demonstrating Results: 
Evaluating Local Out-of-School Time 
Programs (Little, DuPree, & Deich, 
2002). These publications provide use- 
ful information about designing and 
implementing evaluation systems, 
including collecting multiple data 
sources and effectively using stake- 
holder surveys to enrich the evaluation. 
Surveying the parents and teachers of 
children receiving supplemental edu- 
cational services will increase the relia- 
bility and utility of the evaluation. 



Incorporating multiple sources greatly 
increases the accuracy and usefulness 
of the evaluation for stakeholders, 
policymakers, and providers. 



After School/ 
Out-of-School Time 
Evaluation Tools 

When evaluating supplemental service 
providers, it is critical to document 
individual student achievement. How- 
ever, the importance of evaluating a 
provider in a larger context should 
not be overlooked. Many providers 
will come from existing after-school 
and out-of-school time programs, and 
there are excellent evaluation tools 
available in this field. Two examples 
are Northwest Regional Educational 
Laboratory’s Out-of-School Time Pro- 
gram Evaluation: Tools for Action 
(Geiger & Britsch, 2003) and Harvard 



Studies have shown that when 
existing after-school programs seek 
to become supplemental service 
providers, states will recruit better 
applicants if the providers are 
informed about the requirements 
(Flynn, 2002). Faith-based and 
community organizations are eligible 
to become supplemental service 
providers, and they face specific 
requirements. States should encour- 
age these organizations to consult 
the Toolkit for Faith-Based and 
Community Organizations to Pro- 
vide Extra Academic Help available 
at www.ed.gov/admins/comm/ 
suppsvcs/toolkit.pdf. 



Existing Title I 
Monitoring Practices 

Evaluating supplemental service 
providers is just one part of overall 
Title I monitoring. Consequently, states 
should integrate the evaluation of 
providers into existing practices, thus 
saving time and money. For example, 
the SEA is responsible for determining 
if the LEA is in compliance with Tide I. 
While performing this monitoring, the 
state should also make certain that the 
LEA is meeting its responsibilities in 
terms of supplemental service 
providers. State and local education 
agencies must work together to ensure 
that the supplemental service require- 
ments are met. 

The table on pages 4-5 is a sum- 
mary of the basic prompts that states 
can use when designing the summative 
component of the supplemental educa- 
tional service evaluation system. For 
each prompt, states should identify vari- 
ous types of evidence and key indicators 
that will document whether the prompt 
has been met. The columns in the table 
for evidence and indicators have been 
left blank intentionally, except for a few 
examples. States should determine their 
own indicators and standards for 
acceptable evidence. To enrich the eval- 
uation, it is desirable to use other data 
such as parent and student surveys, 
standardized test scores, and assess- 
ments administered by the provider. 
Incorporating multiple sources greatly 
increases the accuracy and usefulness 
of the evaluation for stakeholders, poli- 
cymakers, and providers. 
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Summary 

Evaluation of supplemental educational 
service providers is one of many 
responsibilities that SEAs and LEAs 
now must fulfill under the No Child Left 
Behind Act. Perhaps more important, 
evaluation is a way to determine if 
stakeholder needs are being met. To 
successfully meet this challenge and 
ensure that eligible students are receiv- 
ing high-quality supplemental services, 
SEAs and LEAs should work together, 
collect and analyze data from multiple 
sources, and borrow from the best 
practices of other program evaluations. 

Northwest states have always been 
known for their ability to meet new 
challenges with innovation and 
thoughtfulness. By demonstrating lead- 
ership in evaluating supplemental ser- 
vice providers, the Northwest region 
can inspire others to not just comply 
with the requirements of NCLB, but to 
ensure quality and make the intent of 
NCLB a reality. 
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Resources 

Alaska 

State of Alaska, Department of Education and Early Development 
www.eed.state.ak.us/ 

Title I programs 
www.eed.state.ak.us/tls/migtl/ 

Supplemental educational services, frequendy asked questions 
www.eed.state.ak.us/nclb/sesfaq.html 

Approved list of providers 
www.eed.state.ak.us/nclb/providers.html 

Application for providers 
www.eed.state.ak.us/nclb/pdl/SSP_RFA.pdf 

Schools not meeting AYP and/or with students eligible 
for services 

www.eed.state.ak.us/nclb/silist.html 

www.eed.state.ak.us/nclb/districtchart.html 

Idaho 

Idaho State Department of Education 
www.sde.state.id.us/dept/ 

Title I programs 
www.sde.state.id.us/sasa/ 

Apphcation for providers 

www.sde.state.id.us/sasa/docs/resources/SupplementalServices.pdf 

Approved list of providers 

www.sde.state.id.us/sasa/docs/resources/ 

SupplementalServiceProvidershst.htm 



Schools not meeting ATP and/or with students ehgible for services 
www.sde.state.id.us/sasa/resources.asp#Supplemental 

Tool for evidence of effectiveness for scientifically-based research 
www.sde.state.id.us/sasa/docs/resources/SBR.pdf 

Montana 

Montana Office of Public Instruction 
www.opi.state.mt.us/ 

Title I programs 

www.opi.state.mt.us/Titlel/index.html 
Approved hst of providers 

www.opi.state.mt.us/PDF/Titlel/ApprovedProvider.pdf 

Apphcation for providers 
www.opi.state.mt.us/PDF/Titlel/SupServPacket.pdf 

Oregon 

Oregon Department of Education 
www.ode.state.or.us/ 

No Child Left Behind, Consohdated programs 
www.ode.state.or.us/iasa/ 

Supplemental services information 
www.ode.state.or.us/iasa/supped/index.htm 

Supplemental services provider criteria hst 
www.ode.state.or.us/iasa/supped/providercriteria.pdf 

Approved hst of providers and apphcation (Word 
document available on this Web page) 
www.ode.state.or.us/iasa/supped/index.htm 




Schools not meeting AYP and/or with students eligible 
for services and sample parent notification letters 
www.ode.state.or.us/nclb/ 

Washington 

Office of Superintendent of Public Instruction 
www.kl2.wa.us/ 

Tide I, Learning assistance program, approved list 
of providers, and application for providers (Word 
document available on this Web page) 
www.kl 2 .wa.us/Titlel/default.aspx 

Supplemental educational services information 
(Word document available on this Web page) 
www.kl2.wa.us/TitleI/NCLB.aspx 

Schools not meeting AYP and/or with students eligible 
for services 

www.kl2.wa.us/ESEA/Contact.aspx 

Sample parent notification letters and sample contract 
www.kl 2 .wa.us/ESEA/ModelLetters.aspx 

Federal Resources 

U.S. Department of Education, Cross-site search indexing 
http://search.ed.gov/csi/ 

• This is an excellent resource in searching for resources 
across many federal agencies, including the REL network. 

U.S. Department of Education, NCLB Web site 
www.ed.gov/nclb/landing.jhtml 

U.S. Department of Education, Supplemental educational 
services, Frequently asked questions 
www.ed.gov/parents/schools/choice/choice.html 



Keeping schools open as community learning centers: 
Extending learning in a safe, drug-free environment 
before and after school, by U.S. Department of Education, 
Partnership for Family Involvement in Education 
www.ed.gov/pubs/LearnCenters/index.html 

• A good resource for informing the evaluation of existing 
programs that become supplemental services providers. 

Supplemental educational services: Non-regidatory 
guidance, by U.S. Department of Education 
www.ed.gov/poficy/elsec/guid/suppsvcsguid.doc 

• Released in August 2003, this document is the most com- 
prehensive information on supplemental educational service 
requirements, and addresses many issues not included in this 
Topical Summary, such as funding. 

The program manager’s guide to evaluation, by U.S. 
Department of Health and Human Services, Administration 
for Children & Families 

www.acf.dhhs.gov/programs/core/pubs_reports/prog_mgr.htiul 

• Good program evaluation tool. 

Toolkit for faith-based and community organizations to 
provide extra academic help (supplemental educational 
services) , by U.S. Department of Education 
www.ed.gov/admins/conmi/suppsvcs/toolkit.doc 

National Resources 

Education Commission of the States, No Child Left Behind 
(NCLB) database 

www.ecs.org/htnti/of!site.asp?document= 

http % 3 A% 2 F% 2Fwww%2Eecs%2Eorg%2FNCLBsurvey 

• A searchable database of resources on NCLB. 

Finance Project, Information for decisionmaking 
www.financeprojectinfo.org/ 



Finance Project, Out-of-school time 
www.financeprojectinfo.org/OST/default.asp 

• Excellent source of various resources on after-school 
and out-of-school time programs. 

Harvard Graduate School of Education, Harvard Family 

Research Project 

www.gse.harvard.edu/hfrp/ 

Harvard Family Research Project, out-of-school time 
program evaluation database 
www.gse.harvard.edu/hfrp/projects/afterschool/ 
evaldatabase.html 

• A searchable database of program evaluations — 
worthwhile to inform the design process. 

National Institute on Out-of-School Time 
www.niost.org/ 

Accountability for after-school care: Devising standards 
and measuring adherence to them, by M. Beckett, 

A. Hawken, & A. Jacknowitz, RAND 
www.rand.org/publications/MR/MRl4l 1/ 

• Advocating comprehensive program evaluation. 

Documenting progress and demonstrating results: 
Evaluating local out-of-school time programs, by P. Little, 

S. DuPree, & S. Deich, Harvard Family Research Project 
www.financeprojectinfo.org/Pubhcations/OSTlocalevaluation.pdf 

• An excellent tool for developing comprehensive program 
evaluation. 

From the capital to the classroom: State and federal 
efforts to implement the No Child Left Behind Act, 
by Center for Education Policy 
www.cep-dc.org/pubs/nclb_full_report_jan2003/ 
nclb_full_report_ jan2003.pdf 

Making the case: A fact sheet on children and youth 
in out-of-school time, by Wellesley College, National 
Institute on Out-of-School Time 
www.niost.org/publications/ Factsheet_2003.PDF 



SEA toolkit on supplemental educational services in 
the No Child Left Behind Act of 2001 (Draft Version No. 

1.3), byj. Alexander, M. Hornbeck, C. Stumbo, & 

S.W. Fleischman, Council of Chief State School Officers 
www.ccsso.org/content/pdfs/SSPToolkit.pdf 

• The most comprehensive toolkit on supplemental educa- 
tional services. Excellent process design tools. Draft docu- 
ment, but still very useful. 

Statewide educational accountability under NCLB: Central 
issues arising from an examination of state accountability 
workbooks and U.S. Department of Education reviews 
under No Child Left Behind Act of 2001 , by W.J. Erpenbach, 
E. Forte-Fast, & A. Potts, Council of Chief State School Officers 
www.ccsso.org/publications/details.cfm?PublicationID=215 

Title 1 supplemental educational services and after-school 
programs: Opportunities and challenges, by M. Flynn, 
Finance Project 

www.financeproj ectinfo.org/Pubhcations/suppsvc. pdf 

REL Resources 

Northwest Regional Educational Laboratory, 21st Century 
Community Learning Centers Web page 
www.nwrel.org/ecc/21century/ 

Literacy program evaluation tool, by North Central Regional 

Educational Laboratory 

www.ncrel.org/hteracy/eval/ 

Out-of-school time program evaluation: Tools for action, 
by E. Geiger & B. Britsch, Northwest Regional Educational 
Laboratory 

www.nwrel.org/ecc/21century/pubhcations/ost_tools.pdf 

• An excellent resource and tool for designing comprehen- 
sive evaluation systems. 



• • By demonstrating leadership in evaluating supplemental 
service providers, the Northwest region can inspire others to 
not just comply with the requirements of NCLB, but to ensure 
quality and make the intent of NCLB a reality. ? ? 
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